Skip to content

fix(prose-tests): the stop hook records, and names the model that walked#559

Open
leeovery wants to merge 1 commit into
fix/discussion-handoff-sourcefrom
fix/prose-tests-record-the-model
Open

fix(prose-tests): the stop hook records, and names the model that walked#559
leeovery wants to merge 1 commit into
fix/discussion-handoff-sourcefrom
fix/prose-tests-record-the-model

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Revives a hook that has never fired. prose-walker declares a stop hook, but record-action.cjs scopes every event to a world by finding the world path in the payload — and a stop event carries no tool input, so there was nothing to match. The record was silently short. Its world now resolves from the agent transcript the payload points at.
  • Names the model each walk ran on. The same transcript states the model per message. An edited agent definition doesn't reach a running session until plugins are reloaded, so a walk can run on a model nobody intended and every verdict is then trusted at the wrong one. The model travels from the harness record → asserter → orchestrator verdict block, so a stale definition shows up in the output instead of being something to remember.
  • Hardens the asserter-violation log — it appended to a path whose parent it never created, so a contract breach went unrecorded wherever that directory was absent.
  • Drops a stale claim in the orchestrator description that it escalates failures to Opus; it hasn't since both agents moved to the same model.
  • Adds tests/scripts/test-prose-record-action.cjs (14 tests, registered in npm test) — the recorder had no suite, which is how the dead hook stayed invisible. Covers both world-resolution paths, model extraction (single, multiple, unreadable, partial trailing line), the asserter contract, and the never-break-what-it-observes guarantees.

Test plan

  • node --test tests/scripts/test-prose-record-action.cjs — 14/14
  • node --test tests/scripts/test-prose-cases.cjs tests/scripts/test-prose-snapshots.cjs — 35/35 with the new suite
  • Hook driven end to end with a synthetic SubagentStop: resolves the world from the transcript, extracts the model, tolerates a truncated line, writes one uniform row
  • Verified live against a real walk — the payload's agent_transcript_path proved a walk ran on claude-opus-5 while the definition on disk said sonnet, which is the case this change makes visible

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559 👈 current
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575

The walker has declared a stop hook for weeks and it has never written a
line. Every other event is scoped to its world by finding the world path
in the payload; a stop event carries no tool input, so there was nothing
to match and the record was silently dropped. Nothing failed — the record
was simply short, which is the failure mode this whole subsystem exists
to prevent.

The stop payload does name the agent's own harness transcript, and that
file states the model each message ran on. Resolving the world from the
transcript revives the hook and, in the same read, answers a question the
verdicts could not previously answer: which model produced this result.

That matters because an edited agent definition does not reach a running
session until its plugins are reloaded. A walk can therefore run on a
model nobody intended, and every verdict in that run is trusted at the
wrong one with nothing on the page to show it. The model now travels from
the harness record through the asserter into the orchestrator's verdict
block, so a stale definition is visible in the output instead of being
something to remember.

Also hardens the asserter-violation log — it appends to a path whose
parent it never created, so a contract breach went unrecorded anywhere
the directory was absent — and drops a stale claim in the orchestrator's
description that it escalates failures to Opus, which it has not done
since both agents moved to the same model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant